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This document defines performance testing as the task 
of determining how well performance objectives are mastered by 
learners. Examples are provided for teachers who have to conduct 
performance testing with industrial standards. Stress is laid on 
communicating and exchanging . materials with colleagues. A reporting 
form is included which shows how input material can be entered into a 
CISNE (computerized information system for nursing educators) w This 
computerized curriculum package has implication^ for both state 
syllabi and for individual teachers. Examples are given of how this 
can be done from a technological point ^of view which applies 
behavioral objective and criterion-reference testing theory- 
Flowcharts are presented which sum up these operations of CISNE. 
Since CISNE can produce modules, examples are given of the various 
components of these modules. (Author) 
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PERFORMANCE TESTE SESSlfx 1^ EVALUATION 

in the Difficult Task of 

Determining How Well Objectives Are Attained 



THEIR CONCEPTION 



A Summary Memo on Putting This Respcrch to Work 

PERFORMANCE TESTING : Putting Research to Work 

for the Classroom Teacher Interested in 
Industrial Standards 



THEIR ELABORATION 



ACCEL»S1BILITY : Finding Out What Other Educators 
Are Doing with Goals, Tests, and Resources 

<j 

THE FIRST STEP : How to Start Writing Understandable 
Performance Objectives 

REPORTING FORMAT : The Question with Many Possible ^ 
Answers and Approximations 

THE NEXT STEPS : Starting to Think in Terms of What a 
Computerized System Can Do That an Individual 
Teacher Can't . 



THEIR FORMULATION 



CURRICULUM : State"" Syllabus? or a Tally of What Is 
Actually Taught in the Classrooms across 
' One or More States I 

THE SYSTEM : Tying It All Together : A Few Flowcharts 
Designed to Sketch out the Big Picture of Modules 
That Unify Educational Targets, Teflts, and Technology 



•*What are we trying to do?" ^ 
"How well have we done it?" 
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Teachers luLcrcsccd in Per f ordnance Testing 



FROM:' Howard P. Alvir 
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Putting; This Research to Work 



Khat to Oo 



It you feci pcrformoncc type of testing 
can ir.^vjove your reaching and aid your students, 
led l J^c to ^ct in touch with us at the above 



For a limited number of teachers, it will 
be possible to set up an arrangement that will 
^eC valuable evaluatipn information into their 
har.ds. 



■rhnt Yo-a VJill. V,e Expected to Send 

l" Pv'rf orv.mre and Behavioral Objectives 

For eaca item, you want evaluated on this 
forQatj you will submit a performance obiective 
on the unclosed format, reporting form, #8 

2 - - v" : > r r ' - • p r; i\ ■ i. n Test Items 

/or each objective, you will construct a. 
corft'ccponding test item. Those tests which 
cc/^.M. Lne several objectives and items into a 
:;ia ;iu evaluative instrument will be given 
priority. 

.3--Te£t nar.g 

For each student, you will provide the 
following information in tabular form: 

o student number 

o evaluation of each item (maximum of 
tea itcins per test); this will usually 
be in the form of quality points ranging 

from 0 to 9 

working time in minutes 



o 



o 
o 



o 

o 



V?hat You Will Receive From the System 

o total quality points- for each 
student / 

average working time of the group 
results of Formula I for each studenC 

(peer group standard)^ 
results of Formula II for each stude 

(industrial standard) 
results of Formula lil for each 
student (industrial time standard 
for- maximum quality work only) 
efficiency score (I, il, or III) 
trainee efficiency measured as 
percent of industrial standard 
average group time 
average Efficiency score 
average trainee efficiency measuremen 
as ..percent of industrial standard 
if a retest is desired, all the above 
Is available plus the following 
comparisons (test/retest) for both 
individuals and group percent loss 
or gain by efficiency scores 
o percent loss of gain by 

industrial standards ' 
o statistical significance of 
loss or gain 



More is Available 

The question arises, "Hoxa did the above 
list of outputs develop?'^ 

The answer is simple: Interested 
educators said they would like to have this 
type of information but it took too much 
time. Computer programs were then developed 
to meet this expressed need. 



Jur each batch, you will provide the 

foliov/in^', identification: 
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o instructor code 
o iiehoal code 
0 job code 

o naxiraum number of quality points 

^ '^'2 " Time of industrial standard or 

experienced operator 
0(^2" Industrial standard quality points 

or experienced operator 



If you have questions or if .there is 
other information you want out of this data,^ 
feel free to communicate with the Bureau of 
Occupational Education Research. 

Wait until all is cleared before starting 
to ^end your data. 



3 ' 



, 0- 

/ 

/ 



PERFORMANCE TESTING : Putting Research to Work 

for the Classroom Teacher 
Interested in Industrial Standards 



Many teachers feel that ia combination of performance objectives 
as a rough guide and indui^ trial standards as a measuring tod created a 
team that can improve^ the teaching-learning process* 

The question arises, '^How did svich teamwork develop?" 

The answer is simple : Interested educators said they would 
like to have information available to them providing peer group standar4s, 
industrial quality standards, and industrial time standards but it would 
take too much time for an individual teacher. 

Computer programs were then set up to get this valuable evaluation 
information into the hands of educators. It is a two-way street : the 
teacher receives a helpful service and the researchers receive information 
from the resulting feedback that continually improvides the quality and 
the specificity of the service rendered. 
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Wh.'it the Tonchcr Do"g6 



Performance and Behavioral Oblecttves 



For each item the teacher wants evaluated, 
he submits a performance objective in a format 
vhicli includes : 

o ctoss-rcferenced coding 

o conditions, performance, and c^rteiit 
of the particular objective. 

In actual practice, this includes the 
following previous activities: 

o decisions about what is essential 
and what is incidential in the 
conception of a particular course 
on the local educational agency 
level 

o selection of objectives from lists 
constructed and written by teachers 
' giving similar courses 

o writing and ' cross -referencing only 
that Local objectives which are not 
available elsewhere through use of 
the computer feedback system. 



2-- Cross ^' Ref erenced Teat Items 

For each objective, the teacher will 
construct a corresponding test item. 

Those testa **vhich combine several 
objectives and items into a single evaluatiivc 
instrument would seem to be of high priority 
at this point. 

In actual practice, this includes the 
following activities : 

• going ovet^old examinations, both 
paper-and-pencil and paycliomotor, 
and deciding' whether or not a 
particular item really gives the 
type of information needed by the 

--school, teachers, and trainees 

« if a particular item is valuable, 
then it must be related to an objec- 
tive to establish its priority; 
otherwise, it is removed from' consid- 
eration as a prime concern, 

e often a clearly written performance 
objective will make the t'est item 
obvious or self-evident 

• once test itmes have been pross- 
reterenced with essential and incidcn 

• tal objectives, it will often be 
necessary to construct new test items 
to measure essential objectives that 
are not currently being adequately 
evaluated by existing testing 
^ instruments . 
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Gonl t Mnnr.p.omcnt , N6t Merply Enumeration 



,Rathar thnn become bogged down in 
tntiurnioi.ml:nljlo paperwork, tho toacher ehould 
rcallx;c tfuit his task is to ^atn control* of ■ 
the educational situation. 

At tho moment of entering upon a feedback ^ 
system, the prime consideration is not that 
evory course- hnve every pbssible terminal 
and enabling objective clearly writteit up in 
a file cabinet somewhere. 

\ 

^\ The Important tiling is that teachers begin 
to decide which objectives are really wortli 
teaching find measuring, .Rather than get 
sv;nnipcd in details, it is suggested that 
teachers stai*t with the most important objec- 
tives of their courses. Once these are taken 
care of, the teacher will be able in later 
work to spell out inciden:tal learnings which v. 
are important hut whirb arb. not as essential 
as the prime iiiiportance obje^ives which must * 
be learned by every success fulNtrainee and 
student. " \^ 

A real feedback system will give the 
following capabilities to each participant: 

o the ability to edit, adapt, and 

rewrite objectives to fit new needs 
and new conceptions \ 

o the ability to profit from the 
successful and well written 
objectives and items of others 

o tho ability to revise one's own 
curriculum vithout having to 
choose objectives for which no 
support (clear statement, test 
items, and cross-referenced 
media) exists. 



3-- Test TXffta 

II . ■ !■ ■ ■■11. I V 

Once the participant has de-ctdcd upon 
"his or her ob jectivcs , " i . e . , those objcc- ^ 
tives which have been personally written or 
chosen ftom the work of others , student and 
test data are necessary for a complete ' 
analysis. 

For each student, the teacher will prbvidi 
the following information' in tabular for data 
processing form: 

© student number (the complete 

computerized file is necessary only 
the first tim^) 

® evaluation' of each item (maxim«tn^f^^' 
ten itmc^ per test) ; this will usually; 
be in th~e form of quality points 
ranging. from 0 to 9 

o working time in. minutes. 



For each batch, the teacher will provide 
the following identification: 

9 instructor code 

G school code 

o -Job code 

Q maximum number of quality points 

© -T^ - Time of Industrial standard or 
of experienced operator 

« Qi - Industrial standard quality 
points or of experienced 
' operator 
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Scry 1 CP , Not Monollllilc Siini Inri ty 



Vjh?^t the^Teacher Will Roccive 



Tho above procedures are for those \ 
educators intc^restcd in having data on peer, • 
quality, and time s^tand'ards in the correction 
of examinations. 

Rather 'than appear as a prescription for 
all educational institutions, it is proposed 
as^mercly ^.n example of 'the tailor-made ^ 
options available. 



System , Non Immutable Stability 



The abdvc procedures are for those 
oducaLOTS who realize that their first efforLs 
at wriLlnfj objcctjLves and cross-referenced- 
tost items will not always give immod:*-ate 
results of the hightest possible quality, 

•However, with a systematic approach, 
. teachers and other educators will be able 
to update, edit, and revise both their 
objectives and test items. 

This means that the system will try to 
set up the computerized^ hardware necessary . 
to make this possible. 



For every objective^. and cross-ref erehcei 
test item submitted, the iixstructor could 
receive the followihg^ 

• total quality points for each studen 

o i^average working time of the group 

^ o results of Formula I for each' 
student (peer group standard) 

o results of Formula . 11^ for each 
student (industrial standard) 

• results of Formula III for each 
student (industrial time standard 
fc^r maximum quality work only) 

o efficiency score (I, II, or III). 

« trainee efficiency measured as 
, percent of industrial standard 



Structure , Not TJrH^rocesscd Data 



The above procedures are designed to 
facilitate an intellectual audit* of the 
widest possible scope. 

Rather than say that the following 
courses are taught at a partfcular local 
educational agency, it will be possible to 
spell out with performance objectives exactly 
what is taught under a particular course name. 

In time, it will be possible to spell 
this out in terms of entry level job skills. 



• average group^ time 

• average efficiency score 



■ o average trainee efficiency measureme 
as percent of industrial standard 

• if a retest is desired, all the abov 
is available plus the " f ollov;ing 
comparisons (te'st/retest) for both 
individuals and group percent^ loss 
or gain by efficiency scores 

0 percent loss of gain by industria! 
standards 

• statistical significance of loss 
or gain 

Individual adaptations of the ^bove are 
always available to respond better to local 
needs and goals ^ 



J 



Finding Out 

What Other Educators ^ 

Are Doing 



THE FIRST STEP : Hov to St.art Writing 

. Performance Objectives 



Whether or not . there are a large number of teachers who 
have started writing performance objective^ for their students is 
a ^question open to discussioHr ■ , 

Once one defines & performance objective as a formulation 
an educational goal spelt out in terms of measurable performance 
as its criterion, it is obvious that thousands of teachers have done 
the n^icessary thinking and deciding to develop criterion-reference 
goals for their classroom activities. 

Because much of this work has occurred without feedback 
from other teachers in similar courses, a great amount of educational 
planning has been done over and over again in a way thnt resembles the 
repetitious activities of Sisyphus. 
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The Working , Force for This Pioject 

At the. present time, there are thousands 
of tenchers v^ho arc spending long hours 
prepa^-ing their classes. 

Whether these teauher.s realize it or not, 
all the work they do to develop interesting 
lt»sson plans, well thought out test questions, 
<'^ni practical teaching aids are really part 
of performance objectives* 

They havre done enough work to . provide a 
sound hnsic for educational Innovations, Only 
6nc thing Is missing: they have not provided 
a method of. recording, classifying, and 
evaluating all their work. 

Because this work is repeated over and 
over again without feedback, it is much like 
a book misplaced in the library. For all 
practical purposes,- it is lost. 

The First Stop - Write Only One Objective 

In ordar to get teachers to start v;riting 
performance objectives, it is absolutely 
essential that they do not .write too much. 

The im[)ortant thing is^not to write 
" hundreds of objectives for S particuJ^ar 

courso, but: to write one or two carefully 
^. choeen objectives that stress principal goals 
of a particular teacher ^for a particular 
group of students.,^ ^ 

It would be better for a teacher & 
voluntarily, to wrl,te in half an hour one 
performance objectiv.e which is clearly 
understandable th^n under duress to write 
hundreds of performance objectives which are 
more paperwork th^n practical teaching and 
learning guides; besides, all this needless 
writing would take too much time. 

Format 

Sooner or later, a group of teacheri^will 
want to develop a uniform format on which to 
record and write performance objectives, A ' 
la^T- section gives one of many possible formats 

"ERIC / 



The format eventually chosen should 
reflect two things: 

• local needs 

0 

9 compatability with a feedback systcta, 



Then Pass It Around for Comment 

Once an objective has been written, it 
would be given to colleagues and to students, 

Any difficulty they might experience in 
understanding precisely what he meant would 
force the teacher to get rid of vague words 
such as "know" and "understand," This right! 
critical audience would force the teacher to 
express his objectives in concrete t^^rms. 

For example, if he knew he was going to 
rely ov\ fill-in questions aS his sole 
evaluative criteria, he would have to say 
thaJn^his observable and measurable goal was 
"the ability to recall from memory well 
eaough to answer fill-in questions" instead ' 
of saying that he wanted his students "to 
re^illy know" his subject matter. 

Another example: if he claimed higher 
goalsv such as "creativity" or "deeper 
understanding," he would be forced to ask 
himself, "How am I going to measure these 
outcomes which cannot be ascertained f^i;om 
strictly recall ^questions?" ^ ^''^ 

All of this would have to take place ^ 
before. the course began and not just before 
the final exam. 



An Aid to Evaluation 



A teacher is asked to write merely one 
performance objective , not to give him more 
paperwork, but, to get him to ask concrete 
questions about exactly what he is expecting 
of himself and of fiis students in the 
classroom. 
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Once n ti^achcr puLs his objectives in 
tcrciu; that can bci roco^nizcd by the average 
man on the street, he vill find out tV70 
things: 

o he will get suggestion after 
suggestion on how to improve 
cither his goals or his methods 
or his evaluation tcclmiqucs, and 

0 lie will begin coming up with many 
improvements hi'^l3el f . 

i 

It's similar to a situation wherein one 
starts competing with oneself, it's hard to 
remain satisfied even with above average 
resuL ts . 

This seems to be the motivation in golf 
where— both pro and amateur keep getting better 
or worse becatise there is a very visible 
method of keeping score. 

The Second Step - Correlate One Test Item 
with a Well Written Objective 

Once a teacher has written one performance 
objective, he will know the difference between 
vague and precise terminology. 

The next step would be for the teacher 
to take the best test question off his last 
examination and try to identify which 
objective is being measured by this test item. 

.After this has been done, he would aak 
his colleagues and students to tell him 
whethi»r or not this item is measuring 
understanding or merely memory recall. 

Professional Growth 

It is a revealing experience for a 
teacher to struggle with the uncertainty that' 
arise from a few simple questions: 

o "Which course objective does this 
test item really measure?'* or 

o '*ilow does one construct a way to 
measure the creative goals of this 
course?" 



After a few hours alone with these 
questions, the teacher will realize that 
he needs feedback: he needs to know v;hat 
other teachers of the same and similar 
courses are doing* 

He will then be in a position to learn 
from them their secrete of success to share 
with them what he has learned from personal 
experience « 



The Third Step - Start to Rate and Classify 
Obj ectives 

Once the teacher understands the concrete 
terminology of performance objectives and 
how to relate them to test questions, he 
is ready to start choosing the most important 
objectives of his course. 

These objectives need not necessarily 
be perfectly written and unalterable^ 

Ev«*»n if Imperfect, these objectives will 
allow the teacher to make comparisons and 
revisions . 

Once his objectives are codified by 
subject macter, he will be able to compare 
his objectives with those of other teachers 
of the same material. 



The Fourth Step - Tie Objectives and Tests « 
and Med i a Together into £ System 

Once he starts getting feedback from 
other teachers about their objectives, he 
will be able to use tools they haVe found 
successful. This would include: 

9 teQt it jma , 

e evaluation Instruments, 

• Instructional media, 

• other techniques. 
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Tilo FjJiil Ston - Stinrt Using tjie Performance 
Ob j( c t i V(? Code Kumbor 



In the Sr'imc manner, once a subject matter 
is dividctd into a carefully codified set of 
objoclives, tbo teacher is able to apply the 
«aine coile nunibtTB to teaching aids and other 
educational brainware and hardware. 

Thus, when the teacher has chosen a 
particular objective for his students, he 
will be able to obtain a number of helpful 
teachi.->e, »^ools: 

© teaching aids, 

o classroom projects, 

0 lesson plans , 

o classroom activities, 

o supplementary reading, 

e testing, and 

Q evaluation instructions by means 
of one code number* 

In the same way, with the same code number 
the student will be ^blc to obtain similar 
learning 

o individual assignments according to 
his ability and interest, 

o projects developed especially 
for him, 

o study guides designed with his 
ability and interest, 

o projects developed especially for 
him, 

o study guides designed with his 
ah-ll-lties and strengths in mind, 

e extra tutoring when and where it 
is necessary, and 

c a profile of his marketable skills 
which achievement of a particular 
objective will give him. 

ERLC 



The Sixth Step - Ony>ol ng Acf ivit Jes 

This section has stressed only a 
general overview of the first steps necessary 
to start writing performance objectives in 
a systematic way. 

It takes a lot of time and work to make 
these initial extra efforts for the benefit 
of a system. The next section on the 
benefits of a carefully developed system 
will show a few of the long-range advant^K^^^^ 
of this extra care when a teacher begiTns 
writing. 



^ REPORTING FORMAT : The Question With 

Many Possible Answers 
and Approximations 

/ 

After all the theorizing and deciding is done in a 
school or in a department about to start writing performance objectives, 
one common ques^tlon arises , ''What uniform format will satisfy all of 
our needs 

Typically these needs include various possible uses to 
which the objectives wijl be put : ease of writing, possibility of 
editing, reproduction either by copy machines or by computer print-out, 
coding for retrieval and cross-reference purposes, ability to be used 
for developing a tally and picture of the structure of what is actually 
taught in a particular area, built-in research and evaluation potentials, 
and demands of both teachers and students. 

This makes a lot of decision necessary even before the first 
objectives is ready for writing or entry Into the system. ^ 
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EVALUATION SERVICE CENTER 
FOR OCCUPATIONAL EDUCATION 



Bohavloral Object ivo Report I ng Form 



Code No. 



FiGid of Study 
Major- Group 

Sub Group 

Block ' 
Unit 



I nstruct lonal Time 



No. 



Stato ^ 
Schoo I 



I nstructor 
Fad I I tator 
Levo! 



No. TakI ng ObJ . 



Schoo) Yoar Written 
Behav I ora I Object I ve 



Co-^di tionCs) 



Performance 
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Extent 



Direct Prerequisite UnltCs) 
Direct Dependent Unil (s) 
Exit Behavior (Cl;ieck One) 



Yes C ] No C 3 ^ 
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All Those Forms Rar>tn to Look Alike 

After reviewing a typical behavioral 
or portormcince objective reporting from, 
one can bc^in to think, "Thay ' re all 
pretty much alike/* 

Closer examination will show that 
this is true as far- as conditions, 
performance, and cktent are concerned. 
However, even such cotranon elements as 
subject matter code show many possible 
variations both in code and in theory 
behind the code. 



Don't Get Tied to Non-Essential Details . 



Form #9 is presented merely as an 
jixample of what one evaluation service 
center has done. 

Each of the entries on this form was 
no doubt the subject of serious and 
prolonged discussion. 

Going through the information it 
codifies will serve as a good introduction 
to the basic requirements of the typical 
reporting form. 

An Item by Item Analysis 



Code No . ^ 

Tiiis is a summary in numeric form of the 
following information which specified a 
number of things about the objective: field 
of study, major group, sub-group, block, 
unit, sub-unit number, instructional time, 
direct prerequisites , . direct dependent units, 
exit behavior potential, capability 
classification, and related disciplines; 
also in the local writing domain: state, 
school instructor,, facilitator, level, number 
taking objective, and school year written. 



Field of Study 
Major Group 
Sub-Group 

These refer to the first six digits 
of the USOE subject matter code, ^^^"^ 

Example^ 12.0000 indicates the 
field of study (trades and industry). 

17. 0300 indicates the major group 
(automotive industries). 

17. 0302 indicates the sub-group 
(mechanics) . 

Block 

Unit 

Number 

These refer to the 7lh, 8th, 9th, lOth^ 
11th, and 12th digits added to the basic 
USOE code for subject .jnatter. 

Example: 17.03020400 coul d refer to 
engine; 17.03020600 could refer to brakes; 
17.03020800 could refer to chassis. 

Thus, each sub-group would be broken 
down into blocks (7th and 8th digits). 

Similarly, each block would be broken 
down into units (9th and 10th digits). 

Example: 17.03020406 could refer to 
six-cylinder engines; 17.03020408 could 
refer to eight-cylinder engines. 

V When necessary, further sub-divisions 

would be possible through the use of other 
numbears (11th and 12th digits). 
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CAPABILITY CLASSIFICATION 



Psychomotor 

C D Psychomotor (Chock only !f Dorformance requires a significant amount 
of muscular activity). 

Cognitive 

(Check the one cognitive capability that best describes the mental 
activity involved), 

V 

Knowledge Acquioit Ion 

C H CIJ Knowledge of Specifics 

[3 CU2 Knowledge of Ways & Means of Dealing with Specifics 

Knowledge Application 

'[ 3 C2J Knowledge AppI i cat Ion Without Alteration 
[ 3 C2,2 Knowledge Application With Alteration 

Related Subject Discipline (Check those applicable ) 



M ATHEMATICS 

Elomontary School Mathemat l^cs 

r 3 Basic arithmetic & operations 
f 3 Informal algebra , . 

[ 3 Informal geometry 

Mathemptlcs (7th'& iSth grades ) 

C 3 Applied arithmetic 

n 3 Geometry & measurement 

C 3 Algebra, graphs, problem solving 

Secondary School Mathematics 

[ 3 Algebra (first year) 

C 3 Algebra * (second year) 

C 3 Algebra (third year) 

C 3 Geometry 

C 3 Trigonometry 

C 3 Business Ai Ithmetic 

C 3 Consumer Mathematics 

C 3 Shop Mathematics 

[3 Calculus 

C 3 Computer Mathematics 



OTHER 
(Speci fy) 
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C 3 
C 1 
C 3 

r 3 



SCIENCE 

[] 3 General Science 

Biology 

C 3 General Biology 

C 3 Anatomy 

C 3 Bacteriology 

[ 3 Biochemistry 

C 3 ^ Ecology 

C 3 Nutrition 

C 3 Physiology 

Chemistry 

C 3^%(5eneral Chemistry 
[ 3 Inorganic Chemistry 
C 3 gual I tatlve Analysis 
C 3 QuanlTtative Analysis 
C 3 Materials Science 

Physl cs 

C 3 IGeneral Physics 

C 3 Electricity & Magnetism 

C 3 Electronics* 

C 3 Heat 

3 Mechanics 

[ 3 Nuclear Science 

C 3 Optics 

C 3 So lid State Physics 

[ 3 Sound 



DISPLAY 



state 



Condit ions 



School 

Instructor 

Facilitator 

This Information helps identify the 
source' of the objectives (state, school 
which would include school district in its 
code, instructor who wrote, and facilitator 
who did the first editing). 

LovgI 

This code tries to answer a few simple 
questions: 

• Wliat is the level of the program? 

o pre-vocat ional 
o secondary 
o post-secondary 
o other 

o How many, years long is the program? 

o At^ which year in the program is 
this objective offered? 

Nninber Scekin fi Qb.1 active 

This code tries to estimate how matiy 
students will be exposed to this objective 
from two viewpoints: 

• ^How many students are enrolled 
in the course? 

9 How many students will probably 
be exposed to this objective? 
This latter case will be only 
an estimate in situations where 
production (number of jobs 
entering the school shop) is 
a prime consideration. 

School Year Written 

This information will be used fot 
"^"Ut concerning popularity and longevity 




^"particular objective. 



The circumstances ynder which the ^ 
students are expected to perform constitute 
the conditions of an objective. 

Conditions are expected to clarify . 
such specifics as: 

• Do students have access to standard 
tables and texts or to memory only? 

• What tools must be used: 

9 Will trainees be tested under 

school quiet or industrial stress? 

Each of these conditions obviously ] 

makes its own special demands upon the 

objective to be performed. ^ 

^^^^ 

Perf ormaice 

Performance means doing. The 
performance of an objective is what the 
student is expected to do. 

• This performance must be 
measurable. 

• One of the best ways to assure 
measurability is to insist on 

a observable and easily visible 
performance. 

That is why such indirectly 
observable activities '^know, 
understand, and comprehend'^ 
should be repLaced with more , 
visible verbs such as '^change, 
repair, adjust, cut out and 
assemble.'' ' \ 

Extent t 

The minimum acceptable standard of 
achievement constitutes the extent or * 
level of performance. 

There will be an anticipated difference 
between the functioning of a master mechanic 
and the beat student after a one-year course. 

Certain jobs will require stress on a 
number of factors including quality, errors, 
tolerance, percentages, and time. All of 
these must be spelled out as the criteria 
on which the performance will be* judged 
successful or unsuccessful. 
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Once a coding system has been decided upon^^ It is possible to 

simplify the classification information b^ numeric codes. The following 

information can be inserted into the above tape : 

Sequence Number a temporary number used to keep 

track of inpuL. before keypunching 



Year 
Type 
Number 

Continuation 
State 
City 
School 
Level 

Related Subject Discipline 

Capacity Classification 

Number Taking Objective 

Field 

Group 

Subgroup 

Block 

Unit 

Objective 
Exit Level 
Time 

Prerequisite Unit(8) 

Dependent Unit(s) 

X, 



the year the objective was written 
objective, test item, or media 
first card, second card, third card 
double check for second and third cards 
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DirLij:t: Pro.rcqulsitia UjU t (3) 

This is the answer to the question: 

o Which objective or objectives 
must be successfully completed 
before attempting this new 
unit objective? 

Direct Dependent Unit (s) 
This is the answer to the question; 

' ■> 

o Which objective or objectives * 
' should scq uentially follow tlie 
successful completion of thi-s 
unit objective? 

Exit Behavio r 

This is the answer to the question; 

o Docs successful completion 
of this objective give the 
trainee a marketable skill 
for entry level employment? 

, C/ipablllty Classification 

This section is a check-*li8t in three 
arcJis: psychomotor, cognitive^ dnd related 
subjects*, 

'Psychomotor 

This is an *a4:tempt to answer the question; 

9 Does the accomplisV^ment of this 
objective involve a significant 
amount of muscular activity?^ 

Coj>nitlve 

This Is an attempt to answer the question; 

• Docs this objective stress 
acquisition or application of 
knowledge? 

e In other vords, is this objective 
based more heavily upon memory or 
upon higher mental functions in its 
cognitive component? 
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Belated Subject Discipline 

This is an attempt to answer the question: 

e Where does this objective tie 
into b^sic education as far as 
math or science is concerned? 



Hindsight 

The careful reader of the above / 
description of the typical behavioral or 
performance objective reporting form will 
notice, two missing items; 

• there is no place on the reporting 
form to prescribe how the objective 
in question is to be taught • 

• there is no place on the reporting 
form to prescribe how the objective 
in question iSv to be pleasured « 

The reasois are simple; 

• Rather, than spiccify specific means, 
method, or procedures, a performance* 
objective leaves open a number of 
paths to its accomplishment 

9 Rather than specify specific test 
items to be usgd to measure an 
objective, a performance makes it 
crystal clear by its conditions, 
performance, and extent exactly 
what is to be measured. 

• The emphasis is not on a particular 
thermometer test item which may or 
may not always" function correctly 
button the standard of excellence 
expected of the successful trainee. 

^ • Once excellence has been defined 
in clear cut terms, it should be 
much easier. to achieve and I'ecognize, 



THE NEXT STEPS : Starting to Think in Terms of 

What a Computerized System 
, Can Do That Individual Teacher Can' 



It is almost axiomatic thft teachers have dreamed dreams 
that othp.r teachers have fulfilled in later centuries" or In later 
generations. 

With the aid of modern and existing computer technology, 
the modern teather can see a portion of his dream come true in his 
own lifetime if not in a year or two. 

Going onto an automatic data processing ot computerized 
system will not work miracles, but It will save hours, days, weeks 
and even months of routine clerical tasks that must be done to 
achieve the kind of feedback, that answers the "simple question, 
''Did my teaching accomplish what I set out to do?'' 



' caMPii'r££ £ £ a 11 1 E I M £ n H 

Here is an outline summary of things to be done to computerize the evaluation 
service center, 

(a) Objectiveg Onto Tape 

— format of^printout o Code 

o Condition 
o Performance 
o Extent 

" ability to oditf content — proofread and revise content 

ability to add other classifiers 

Ammerson Melching 
other^ needed 

— restructure USOE and content codes ' > ; ^ i 

(b) 

Cross-reference of Test Items • ^\ 

■ ' • M 

" Code ^1 

Ml 

Different forms of same test ^ 
" Levels of difficulty 
Alternate forms 

(c) Immediate Statistics 

Time Standards Option 

(Formulas I, II, III) ^ 

-- Test Correction Option n 

Profiles Individual, Class, Teacher, Course Program, School District, 
State 

(d) Item Analysis (Exactly What Will Be Analyzed) 
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Decisions to Mnke 

In ^'ome ways, the transition from 
dreams to reality comes naturally and when 
one starts to make decisions. 

There are many areas to think over* 
each of which requires an amount of 
specificity that takes time to achieve: 

o how to get the first versions of 
the performance^ objectives onto 
comput et - read.able format 

9 how to cross - re fcrence the test 
items and other media support 
in a way that will facilitate 
both entry and retrieval in a 
variety of formats 

© how to articulate the entire 

system in such a way as to 
^- provide immediately^ necessary 
stnti s tics and lonpfr range 
information needs 

• f • 

o ^how.to decide the type of item 
analysis required by the local 
educational agency and consistent 
with irs budgetary limitations 



o how to decide which student 
chs racteristics are necessary 
^ for later research and correlation 

studies to determine differential 
effects of various objectives 

o how to decide on which ongoing 

activities will receive priority 
t or long-range status* 

Each one of these decisions must be 
broken down into a number of specifics. 

The following sections will look at 
each of them in turn. 



From Paper - and - Pencil to Compu t v r - Road ah le 

Consideration must be given to two* 
types of format: 

e the format filled out by participatin 
teachers 9 educators, and at'l^ other 
sources of objectives 

9 the format of the computer print-out 
which, must be mass-distributed and 
frequently referred to in the 
editing and selection f)rocess 

\ 

This will facilitate a number of things 
necessary to keep the item artd objective 
bank active and yp-to-date: ^^^^-^'^ 

e edltinj (major changes) 

• revisions (minor changt^s) 

o additions' (new entries^ 

e updating (constant renewal) 

9 broadening (humanistic goals). 

Software Considerations 

Some questions will be on a rather 
Ttiundane level : 

o Should ve use straight keypunching 
or a combination of other methods 
tc enter corrected data? 

o Should this be done by physically 
transporting the object Ives .or by 
a computer terminal and a long- 
distdnce phone call? 

© How can cost and system requirements 
be optimally balanced? 

. • How tnuch confidence can we place 
in students, teachers, and 
administrators who will provide 
the balk of the, input' data? Must 
they be constantly supervised 
every step of the way? 
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Ba.*d c C r o s s - Re f c r o r . c e Categories 

Despite individual school needs, most 
local educational agencies will find it * 
convenient to employ three general areas 
of cross-reference: 

© short-range general descriptors 

o subject matter classifiers 

o long-range sequencing indices 

3 

Vcrbnl or Numeric Descrlpt^^rs 



Verbal descriptors refer, to words or 
abbreviations employing the normal alphabet. 

Numeric dcscripbors are a combination 
of numbers, letters, and symbols which are 
not usually derived from everyday language. 

As a practical norm, it is safe to 
f^r y th^t both v.'ill be u£ed by tnoGt coded 
cross-reference systems: 



vorl^al descript£>rs will be 

used, by teachers and supervisors 

as they write objective^ 

numeric descriptors will be 
u^cd by editors and systems 
specialists who prepare th'^^ 
objectives and items for \ 
entry into computer-readable 1 
jEormat 



Illustration ( Refer to Sample on Left ) 



Rather than devote unnecessary 
computer r—imory core to names of states, 
schools, cind fields of curriculum, all 
these areas will be covered by appropriate 
numeric codes which can be entered into 
—code number structures such as that 
illustrated on the left. 
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Examples ; Typical General D escriptors 
These would include such things as: 

© state 

<» city 
1 o school 

9 year objecflive was written 

9 teacher or teachers who write it . 



Subject Matter 
Examples : Identification 

Subject matter or teaching unit 
identification would be based u^)on such 
things as the USOE code: « 

^ , \ ^ 

0 field (1st two digits; "e-. g. 17^ 0000 
would indicate Trades and Industry) 

o Rroup '(3rd aiVd 4th digits; e.g. 

17.0300 would Indicate Automotive 
^ In^-^ustries) 

• subgroup (5th and 6th digits; e.g. 

17.0301 would indicate Body and 
Fende^r) 

o b lock (7th* and, 8th digits; e.g. 

17.03010300 could indicate Body Repair) 

0 ' unit (9th and 10th digits; e.g. 

17.03010307 could indicate estimating 
^cos ts) 

There are alao other considerations: 

• 6xlt (whether or not entry level 
Job skill) 

« level (typ^e and leVel where offered 
in school program) 

• time (amount of instructional time - 
envisioned to teach objective) 

• capability ^ -classification (cognitive, 
affective, or psychomotor domain).. 

number taking objective (how many^ 
students seeking this goal) 

• fP^^Qt^d subje ct discipline (related 
natural science or math background)' 



■ Examples ; ScguonLlal InfoirmaLlon 

^ This Is an attumpt to determine two 

things 

Q direct prerequisite units* or 
objectives necessary 

o 

o direct dependent or sequential 
ohinr.fiVQ© After prpBont goal 

This will help determine which elements 
of the curriculum follow each other from two 
points of view: 

o whnt is actually taught in" a 
particular order by teachers 

o what is actually learned in 
terms of student acheivement 
over a period of time 

o which goals must be achieved to 
, succeed on a particular objective 

o which goals arc appropriate for a 
student who completes a sequence 
before, with, or after the majority 
of other students 



Cross - Re fercncinR Is Never Left Unrevised 

The constant necessity to cross-reference 
and to tctrivc information on a variety of 
Vflriahlcs demands the ability to add other ■ 
classifiers as the need might arisp. ♦ 

o This could occur when a new 
break-through occurs in 
developing a successful 
classification system by 
a toatri of basic researchers. 

o This could occur when another 
• ^ :bank of objectives is pLiced 

en masse into an existing 

classification. 

: Such cpdes as the USOE systdtn (six digits) 
will undoubtedly be used in the initial input. 
Wlion. such codes ar^ found inadequate in their 
seventh' to tenth digits, it will be necessary ^ 
to change the affected code digits. For the 
first time, data will, be available with 
,.ui^u j^ai^e these necessary corrections. 
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I mmediate Statistical Needs 

These .needs can assume a variety of 
foTTms . 

When a teacher starts working in the 
area of performance, he or she will find 
that certain theoretical bases of trainee 
success will assume different forms: 

A the normal distr j butloB g^^ocx^ will 
not always be appropriate for 
measuring the success of students 
on objectives which every student 
is expected to achieve 100 per cent 

0 a distribution gtf Uainail^.. tim^ i-S • • 
to be expected; this will give rise 
to the following situations: 

• teachers will need additional 
later skill objectl^v^s or 
additional higher skills to 
correspond with variations in 
trainee learning time 

a if learning time remait.fi a constapt, 
then each objective v have to 
ha^c a number of lovci'3> af 
accomplishment from basic minimum 
to more advanced skills 

Rather than rely upon traditional pretest 
and post-test measurements, the accomplishment 
of performance objecti-res requires the stress 
upon growth or gains rather tf^n relative 
position on a normal curve* 

e the measurement of gaina will be 
difficult from a normal pretest 
situation since it could be physically 
dangerous to pretest trainees on 

y certain heavy equipment 

, • the actuarial moasuremont of gains 
would also be necessary s ince not ^ 
all trainees would be expected to 
grow or change at the same rate 

% the measurement of gains will 
probably center upon periods of 
time within the structure of a 
, course; e.g. tests will be given 
when a trainee* is safely abl_e to 
operate a* machine on his ov;n as' a 
quasi-prete3t to be compared with a 
post-test measuring the effect of 
additional instruction 
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ITEM ANALYSIS 



o The evaluation service center will develop a number of Objectives classified 
on a variety of categories. 



o Each objective will have a unique identifier: 

This will be a code number which includes among other things: 

• USOE code number (subject matter) 

o domain (cognitive, affective psychomotor) 

0 other cross-identifiers and cross-classifiers 

© After the first' year of testing, each objective will have corresponding test items 
with coded cross-references based on a number of things with the 
ability to make the items more individualized and more accessible : 

• level of difficulty 

e predictability on next sequencial objective 

© Guttman Scale 

• success group correlated (with student characteristics) 
o gains scores ^ 

• list of entry level occupational skills per child 



SUMMARY 



One Unit 



1 



Code 



Objective-Item 



Statistics 



Media 



. Prescriptions 



etc . 



4 

DISPLAY* 
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H>: ample s of ross Iblo Statistical Servicee 

The limits of the human Imagination 
seem to the principal limitations of the 
statistical poasibilltlGs of computerized 
dnta services. 

ilerc arc a few of the many service 
packages that come to mlndi 

• ^ ^ l^^ standards opt J on vhcreby the 
teacher could receive pretest and 
post-test information in^three 
different configurations : 

1 9 peer group standards for 
quality and time 

11 9 Industrial standard for 
quality and time 

ill o Industrial time standard 
for maximum quality only 

^ ^^st correction option whereby the 
teacher could submit the answer cards 
or sheets of the students via an on- 
line computer telephone termin^il and 
receive the results a few minutes 
later through the same terminal 

* s truden t profile options are available 
for a number of things such as 

^ attendance reports, student report 
cards, automatically updated student 
records and transcripts ^ as well as* 
profiles of progress along such 
dimensions as: 

o Individual student 

o f^lass average 

o teacher load 

o course program 

o individual echoci 

o department 

o school district 

e State 



Item Analysis 

Once each objective is coded, it will 
be possible to develop cross-references with 
such variables as: 



• .level (level of difficulty, level 
of previous training, level where 
the objective will be offered in 
school program) 

predictability (trying to find where 
future successes are highly probable 
as a result of success on a given 
objective) 

• Guttman Scale (one of the many 
attempts to scale items with a 
corresponding level of skill) 



e Success ^rou p cor relation (trying to 
find out which groups or which student 
characteristics are highly correlated - 
with success on particular items) 

• Rains scores (trying. to develop 
actuarial tables along a variety 
of cross-referenced student and 
program variables) 

^ entry level occupational skills 
(trying to establish interfacing 
of classfooin objectives with such 
things as computerized Job Banks 
and Inventories of marketable 
skills) 

Item analysis is not an end in itself, 
but it Is a good Instrument of action once 
a local educational agency has decided what to 
do with all the decision making data it 
has collected. 
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STUDENT RECORD VARIABLES 
Each student will have a unique Identifier based upon* 



9 State 

0 School District ' ^ 

o School 

0 Date of birth 

o Social security number 

o Intellectual ability 

© I.Q. 

o Class rank 

o Other test scores — — - indicates areas to be further analyzed 

9 Physical condition 

o Good health 
e Handicapped 
0 Attendance record 

■ ♦ 

o Socio-economic 

Family 
Parents 
Siblings 
Disadvantaged 
Minority group membership 
Career goals: 1. undecided 2, definite 
Drop-out potential 

ecord 



Ml 



O 

o 

& 

o 
o 



o Learning r 



o Class rank decile 

o Track in school 

o Disciplinary 

o High school grades 

o High school grade point average 

o Occupational record 

o Field 

o Time in field 

9 Grades in field 

o Number of courses 

• Part ••time Job 



grade average in occupational area 



• Related to field 
e Non-related 



SWIMARY 
One Unit: 




Student I.D. 



Background 



Performance 



S tuclonL Records 



At the present time, there are a number 
of automatic data processing services that 
provide attendance reports, student records 
and transcripts by the simple act pf pressing 
a computer button. 

Each of these services is Justified on 
its own merits. 

The idea of this section is to suggest 
tryinj] such existing information units 
together into a compatible system. 



Why r.othcr to Day clop Such £ C omplication 

An example easily comes to mind. The 
Vocational Education Act of 1963 was aware 
of the needs of students with academic and 
socioeconomic handicaps. The 1968 Amendments 
made the disadvantaged a top priority item 
by requiring that 157o of the basic Federal 
allotment be used for th^^m. 

There are many ways to identify the 
disadvantaged; not all of them are 
satisfactory but here aria a few of them: 

9 academic disadvantage 

• mental ability (l.Q*) 

o lack of school success (doing 
poorly or failing) 

o dissatisfaction with school 

a irregular attendance 

• soc-lo - economic disadvantage 

o poverty 

o occupation of parents, 

o disadvantaged neighborhood 
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o cultural dtsadvantaae 



9 


family size 


0 


migration 


9 


parents ' education 


9 


marital status of parents 


9 


minority group membership 


9 


neglect 


O 


linguistic isolation 


other disadvantages 


9 


physical ^ 


0 


emotional 


9 


delinquency. 



It is obvious, when one considers such 
factors as minority group membership and 
parents' education, that such potential 
sources of disadvantage do not always 
irreparably harm each individual even 
though they may work against some. 

The Practical Concern Here 



The point under discussion when one tries 
to define disadvantaged in concrete terms is 
not whether or not a particular item or a 
combination of tiems is an adequate 
definition- 

The more general area of concern is this'; 

9 Can objectives be analyzed in 
such a way that various student 
characteristics will be found 
correlated with success on 
these objectives? 

i 

9 Can all of this be 'economically i 
done by means of computerized data 
systems that will merge as needed 
files on test data and student 
records that have up to the 
present been kept separate? 
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T. oo kin^ AlK'.nd - Onp.olng Act tvl tics 

Once a teacher starts usinc, 0 system 
of performance objectives, related test 
items, and other cross-referenced 
educational support tools, he soons finds 
out that He wants more. 

Specifically, these usual demands 
touch the following general areas: 



TEST ITEMS 



® niore test items (alternatives and 

nev approaches to avoid the temptation 
to teaching to the "same old examination" 



• item analysis (replacement of defective 
items and development of basic statistice 
on such things as level of difficulty* 
or discrimination) 



OBJECTIVES 



o revisions (fniror changes in wotds, 
contents , and ' nuances) 

* "^^"^ ^ ^ i^"^ A (utajot* changes In ideas, 
conditions, performances, and 
extent, and objective specificity) - 

« additions (adding nicre objectives 
to fill in missing areas in 
subject matter and curriculum) 



• updating (replacing outdated 
objectives as the state of the 
world of labor demands newer 
techniques or newer procedures 
for entry level employment) 

• brcndcning (adding more objectives 
to fill in necessary training 
goals by due attention to a balance 
of cognitive, affective, and 
p«ychomotor goals needed by 
trainees) 
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CURRICULUM 

o auditing (finding out what is actually 
taught under the same course names 
across one of several states) 



• gtructuring (establishing the order of 
objectives based upon research on 
differential student achievement) 

9 



INDIVIDUALIZATION .OF INSTRUCTION 



% assessment Xgiving a student or trainee 
an idea of where he is in terms of 
objectives accomplished in relation to 
entry level skills, to startling salary, 
and to style of learning) 



• direction (giving a student or trainee 
some specific idea of what he can do 
next with great probability of success 
and of where this will lead him) . 



• tailor - made objectives (showing a 

student where people with his type of 
achievement and personal traits have 
Succeeded) 



CURRICULUM : State Syllabus ? 

or a Tally of What Is Actually Taught 

in the Classrooms across One or More States 



On the level of existing printed syllnbue, there is little 
doubt about what is being taught across a state. Fields of study are 
broken down into major groups, minor groups, and blocks* On the 
local level, these blocks are broken down into courses, units, and 
day-by-day classroom instruction. 

Because the generality of the state syllabus Is designed for 
a wide audience and because the specificity of the local implementation 
is adapted to a more specialized clientele and Job market, it is no 
great surprise to find a different point of view when tallies of . 
performance objectives are compared with state curriculum guides. 
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. JUK PRECTSK KATURH OF A SrATEWIDE SYT.LABUS THE VORK LOAD OF THE CLASSROOM TEACHER 

' A nolincatlon of Both Its Strengths and There Is Only So Much One Person Can Do 

Limitations for Occupational Education ^ No Matter How Important the Priority Is 



A state syllabus or a statewide 
curriculum guide is a multi-faceted tool: 

• Often, it is the result of a 
committee of the most prominent 
teachers chosen across- a state, 

0 This means that much of the 
committee time must be spent 
defending loaal interests and 
making sure they are represented 
in the final document. 

; o Limitations of time and printing 
^ budr;ct moan that certain' local 

programs that^could be successful 
in half a dozen or more other 
similar communities must be left' 
out because they will not fit into 
the patterns of thirty or forty 
"more typical situations „ 



The clacrroom teacher has no shortage 
of things he must do: 

o he must be in the school from 
eight until four; his so-called 
free periods can he eaten up by: 

• prefeeting assignments ^^'^"^ 
» papers to correct 

e 'noi-se in the school 

© administrative paperwork 

© a variety of stressful priorltiee 

• he must be with students twenty-five 
periods or more a week 

o this requires his full attention 
while in the classroom each day 

• this requires adequate lesson 
planning before coming to class 
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o Since the. same objectives must be 
made to work in a school which 
offers fifty or niore occupatipnal 
programs and in a school which 
specializes only in onej^ there 
will be a certain amount of 
anticipated abstraction which 
must be adapted by local 
educational agencieSo 

o When educational leaders get 

together to put out a curriculum 
guide, it can be assumed that it 
will incorporate the latest 
advances available at the time of 
writing. 

e One disadvantage of this approach: 
States of technology will not sit 
still until the. next curriculum 
revision. 



Another disadvantage: '*Ne-too*V 
schools which lay out a lot of 
money for a 1970 shop^ in 1971 will 
be little inclined to spend more 
money to instfall a ^1972 version until' 
the 1973 version is recommended in the 
1975 revision of the curriculum guide; 
the time-lag gap still persists. 



Confrontation with the State Curriculum \ 

There are a number of ways in which; 
the classroom teacher can be given 
additional work because of the State 
syllabus : 

0 his students must pass exams 
based upon common objectives 

• his lesson plans must be written 
in such a way as to make obvious 
how they implement the curriculum,, 

© his supervisor is often more 
familiar with state demands and 
criteria than with the objectives 
chosen or derived by the individual 
classroom teacher himself 

• his colleagues will sometimes tell 
him, ''Even if it's out-dated, tea&h' 

it sincfe it will be in thefinal 

exratnoT' on "tlie" inspect ion report.*' 

Such a framework is the opposite 
of the one herein proposed. This 
proposed structure is intended to 
shift the teacher's role from that 
of spectator to that of participant. 
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Whore Do All These Difficulties Come From? 



Trying to Track . Down Answers 



Rather thnn go around suspecting the 
motivation of curriculum \^ritcrs on the 
State or the local level, it might be a 
good idea to admit a very well known but 
linpublicizcd fact : 



Because of the format of educational 
objectives, it is virtually impossible 
at the present time to obtain data 
which will show how t'he State 
curriculum is implemented on the 
local level. 



Basically, this points to a preliminai 
question that must be answered: 



How do local schools continue 
the classification of subject 
matter beyond the USOE 
categories of Field of Study, 
W^Jor Group, and Sub -Group? 



This results in at least three 
specific information lacks: 



o A tally of exactly what 
portions, of a State 
curriculum are implemented 
in classroom objectives. 

o A tally of local optiona 
that have been proven 
successful and practical 
even though not mentioned 
in the official guides. 

o An estimate , based upon 
the results of actual 
clnssroom performance 
objective testing, of 
what- per cent of cither 
statewide or local 
objectives attempted are 
actually learned by students 
and trainees in occupational 
programs. 

Because the labor market must 
identify products of occupational 
education by documentation (e^g, 
high school diplomas, certificates, 
A.A» degrees, and other licenses) 
which reflects merely statewide 
objectives and not local options, 
it is not always easy to compare 
educational output with job demands* 



Note: U50E Codes 



For those readers not familiar with 
the USOE classification into field of 
study (e. g. , V7.0000 refers to Trades and. 
Industry), major g;roup (e.g., 17^0100 refet 
to Alr'-Conditiontng) , and sub - proup j&ergVr* 
17.0101 refers^ to Cooling), the next three 
pages have'been included to familiarize 
themselves with the overall coding scheme 
for occupational education. 



The Next Step 



Once this information has been 
obtained, it will become obvious that 
certain blocks are mentioned by a number 
of schools. This lead to two distinct 
questions : 

• Which blocks are mentTioned by 
a vast majority of schools as 
being typical components of 
its courses under a given 
USOE code? 

• Which blockis are mentioned only 
by one school or only by a few 
Bchools? 



OFFICE OF EPUCATION CODES 



AGRICULTURE 
(01 .0000) 

01 .OU§p Agricultural -Production 
01.0200 Agricultural Supplies 

County Agent-Farm lnr.uranCG Adjuster 
01.0300 Agricultural' Mechanics 
01.0400 Agricultural Products 
01 .0500 Ornomonta I Horticulture 
01 ,0600 .Agricultural Resources 
01.0700 Forestry 

01.9900 Other Agricultural (Specify) 
Teacher-Farm Exam Officer 



DISTRIBUTION & MARKETING 
(64.0000) 

04.0000 Marketing (General) , 4 

04.0500 Florlstry - Horticultural Merchandizing 
04.0600 Food Distribution. 

O4.0800j5 General Merchandize Management (Gen. /Ml s) 

Mid Management 
04.2000 Retailing (Gen. /Mis.) N.E.C. 

Fash Ion Retal I I ng ^ , 

04.9900 Other Instructional Programs (Specl'y) 

HEALTH OCCUPATIONS 

(07.0000) ' 

07.0100 Dental Services 

07.0101 Dental Asst. 

07.0103 Dental Lab. Tech. 

07.0199 Dental , Other ' 
07.020Q Medical Services 

07.0203 Medical Lab. Asst. 

07.0299. Medical Services (Other) 
07.0300 Nursing 

07.0502 Practical (Voc.) Nursing 
^ 07.0399 Nursing (Other) v ' 
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HOME ECONOMICS - , " - 

09.0100 Homemaking - Preparation for Personal, Home and Family Living 

09.0101 Comprehensive Homemaking or Home Economics 

09.1002 ChlI'd Development 

. 09.0103 Clothing and Textl les ^ 

09. 01 04 Consumer Education 

09.0105 Fami ly Health ' ' ' 

09.0106 FamI I y Relations _ ^ 

09.0107 Foods and Nutrition 

09.0108 Home Management . ,j 

09.0109 Housing and HOme Furnishing 
09.0:199 Other Homemaking (Specify) 
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09.0200 Occupat-ionol Preparation 

09.0201 Care and Guidance of Children 

09.0202 Clothing Management, Production & Services 

09.0203 Food Management, Production &. Services 

09.0204 Home Furnishings, Equipment & Services 

09.0205 Instltu+ional and Homo Managprnont & Supporting Sorvlcos 
09.0299 Other Occupational Preparation (Specify) 



14.0100 
14.0200 
14.0300 
14.0700 
! 4. 9900 



OFFICE OCCUPATIONS , 
(14.0000) 

Accounting & Computing 
Business Data Processing Systems 

Filing, Office Machines, Gen. Office Clerical and Typing 
Stenographic, Secretarial and Related 
Other (Specify) ' 

TECHNICAL 



16.0100 



(Building Construction) 



Fnginoering Related Tech. 

16.0102 Agricultural, Tech. 

16.0103 Architectural Tech. 

16.0106 Civl I Tech. 

16.0107 Electrical Tech. 

16.0108 E'lectronlcs Tech. 

16.0109 Electro-Mechanical Tech. 
16.01 I I Industrial Tech. 

16.0112 instrumentation Tech. 

16.0113 Mechanical Tech. , ■ 

16.01 14 Metal lurgical Tech. 
16.0117 Scientific Data Processing- 

16.0199 Other Relate^ - Op,tIcs Tech., Welding Tech, 

TRADES & INDUSTRY ... 
(17.0000) 



17.0100 



17.0200 



17.0300 



7.0700 
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Air Conditioning 
17.0101 Cooling 
I7.0l0g Heating 

17.0103 Ventilating ( f I I ter i ng & humi d I f I cat I on) 

17. 0199 Other Air Conditioning 

App I I ance. Repa \r" 

17.0201 Electrical Appliance 

Automotive industries 

17.0301 _ Body & Fender - 

17.0302 ' Mechanics 

17.0399 Other Automotive I ndustrles . (Spec I fy) 

Commercial Art Occ. 

J 7.070 1 interior Decorating 

l'fro7r»2 Window Display 

17.0703 Product Des.Ign 

17.0799 Commercial Art Occ. - Other 
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17,1000 Construction (5. Maintenance Trades 

I 7 JOOl, Ca.rp?3ntry 

17 J002 ElecfF?CIty 

17.1005 Palntl.ng & Decorating 

17,1007 Plumbing & Plpeflttlng 

17.1099 Construction & Maintenance - Other 
17.1300 Drafting. Occupation 
17.1400 Electrical Occupations 

* I7.K.0I Industrial Electrician 

17-1499 Other Electrical Occ. 
17.1500 Electronics Occ. 

I V. 1 50 1 Communi cat Ions 

I 7. 1502 Industrial 

17.1599 Other Electronics Occ, (Specify) 
17. 1900 Graphic Arts Occ. 

17.1901 Composition, Makeup & Typesetting 

17.1902 Printing Press Occ. 

I7.!903 Lithography, Photography & Platemaklng 

17.1904 Photoengraving 

17.1905 Silk Screening Making, & Printing 
17.1999 Graphic Arts - Other Offset 

17.2300 Metalworking Occ. 

17.2302 Machine Shop 
. ° 17.2303 Machine Tool Operation 

17.2305 Sheet Metal 

17.2306 Woldlng & Cutting 

^7.2399 Other Meta I work! n9>0cc. '(Specify) 

Precision Sheet MetaSv. 

Mota 1 Fabrl cat ton ^\ 
1 7. 2600 Persona I Serv i ces 

I 7 . 2602 Cosmeto | ogy 

17.2699 Other Personal Services (Specify).' s 
17.2800 Publ ic Service 

17.2801 Fireman Training 

17.2899 Other Public Services (Specify) 
17.2900 Quantity Food Occ. * ^ 

17.2901 Baker ^ ^ 

17.2902 Cook/Chef - " 
17.2904 Waiter/Waitress ^ ^ 
17.2999 Quantity Food Occ. - Otheru^^ 

17.3100 Small Engine Repair (Internal Combustion) 

17.3300 Textile Production & Fabrication 

17.3301 Dressmaking s» 
17.3399 Other Textile Production & Fabrication - (Specify) 

17.3400 LeatherworkI ng v 

17.3401 . Shoe Manufacturing 

17.3402 Shoe Repair . • 
17,3600 Woodworking Occ. ' 

17.3601 Ml I Iwork & Cabinet-making 
17.3699 Other Woodworking Occ« (Specify) 

PatternmakI ng * . " 

House Carpentry , ^ 
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TATJ.Y S1IKKT5 

Putti the ' Ansvors to T he so Questions 

V^^^^'^^ in Qc cupationnl Education 
Planning nncj Awareness On £ Statewide Baols 



Here are the results expressed in 
blocks taught and In frequency of schools 
teaching a particular block. 



Block 



Results of S urvey inf^ Nine Schools 



Each school was a8ke4 to break down 
I'ts major blocks under the USOE code of 
17.0302: 



• 12-0000 — Trades and Industry 



17,0300 — Automotive Industries 



o 17.0'302 ~ Mechanics 



It is Lo be remembered that by 
definition a block is a furt-her subdivision 
of the USOE code to the 7th and 8th digit. 



^ It must also be remembered that, 
as of 1971, no official coding system 
exists for the 7th and 8th digits even 
though each state uses tta own coding 
system for internal repo]:ting or for 
"reporting to USOE. 



The same cd'nstratnts must be applied 
to all tentative attempts' to Specify 
exactly what is meant by the 9th and 10th 
digits which extend blocks into uaits 
of proposed classifying systems. 

The problem of local retrieval 
adds another dimensipn to the problem. 
Since each srhool has its own course 
luimbers, these might well be considered 
as identifiers which would h^ave an 
immense local aeception and practical 
value bsy expressing the system in 
-^^-B and formats already in JLocai use. 



Engine 

Electrical 

Brakes'" 

D r ive (Po\v e r ) Tra in 
Chassis 

Cooling 
Body 

Suspension 
Lubrication 
Tune -up 



Frequency 



8 
8 
6 
6 
5 

4 

3 

3 

2 ' 

3 
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Steering ' ■ 2 

Carburetion 2 . 

Exhaust Service 2 
STP Transmission Service 2 

Special Needa 2 

Rear /Scle Service 2 - , 

Air-Condi t toning 2 ^ 

Alignment 
Alternators 
GeneraCtors 
Electrical Motors 
Tire Service 
Accessories 
Transmission Service 
Shop Orientation 
Engine Overhaul 
Testlng/Hcasuririg 



The above data is merely illustory* 
and i:s not intended to be definitive/ 
It is based on a small sample, none schools 
only, ' 

The above data is ,ni£ixely illustratory 
and- is not intended to be definitive. It 
is based on a small 'sample of only nine _ 
schools-^ pasual inspection of its 
'contents is all chat is needed for the 
purpose it serves in this leection. It 
does show that a tally of actual course 
blocks is 'a simple statistical concept. 



0' 
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ANAT;YSTS 

This Surve y Is 

Not Proposed as a Model of Da t a 
■> 

But as £ Model of DGciyqion ' -Making 



Such a rapid survey will undoubtedly 
suffer from many sources of error: 

© a sample size of nine schools is 
large enough for only a hasty 
overview 

(9 not all possibilities will be 
represented in such a limited 
sampling 

• it was intended to provoke discussion 
and inquiry rather than provide 
definitive data 



Here arc a few of the many actiy^ties 
that resulted from it: 

e It became obvious thiJt only five 
areas (engine,' electrical, brakes, 
drive train, chassis) would provid^ 
a great deal of feedback in comparing 
and contrasting program objective^. 



Vlius, rather\than spend a lot of 
time subdividing such areas as/ 
tire service and\accessorie8 
taught only by one school, it 
was decided to concentrate on 
the areas with the greatest ^ 
amount of possible feedback. 

Similarly, schools which had only 
a few areas began to ask themselves 
whj a particular area was not part 
of th.eir approach, 

This didn't mean that dll schools ' 
would adopt uniforrri programs, but 
it did mean that. local options 
were chosen by teachers fully 
aware of other .local; options 
quite different in scope and 
conception. 
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TyJ.nR Data to Decisions 



Once thn data on classroom objectives 
was in manageable form, it was natural for 
a lot of questions to be asked ^md for a 
lot of comparisons to be madlet 

• Some began to ask why engine and 
tune-up were considered to be 
different blocks and not part of 
each other? 

• . So.e ...a „H,\h.,..e .„ ' 
app«^rent difference between such 
things as chassis and body? 
between STP transmission and 
t?:ansmission service? between 
colling and air-conditioning? 
between electrical and alternatpre 
and generator©? 

After a while, the questions 
shifted from terminology knd 
classification to the more 
fundamental question: 

* 

d Exact^ly what is taught in 
a particular school uader 
each Jot these blocks? 

0 In other words, how does 
your school break down its 
blocks into units? 



Going Beyond Terminology and Provicialism 

Eajch state and each school has its 
own terminology and its own hierarchy of 
course objectives . 

One advantage of such a tally as 
that examined above is that it allows 
good* teachers, even xJhen frotft different 
states, to start working together. 



Note; 

Such a survey will require additional 
rigor and design to make it a source of 
actual block structure. 

/ However J. it is proposed here merely 

as a vehicle to initiate the kind of 
thinking^ that jnust precede sending out 
the survey instruments. 
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SKfT.T.S AK'n Ar.TiJ.TY PATTERNS 

"Kr.QuruKD IN Ti'ir, markht place 

Mnki np, Siiro That £ T.ocal Jld-crgatfton Afi'Wcy 
Doi iL RcnKirknblo J>Db i n Achtcvin,fi 
1)^0 r.j rod SLudrnt: Bor f ormnnc c . Oh jo ct Ives 
Il«i.s Rolcvnncc : to the Job . Market 



This ago-old dilemma in occupation 
education is not a problem that can be . , 
solved by comparing a computer printout ^""^ 
of 10,000 performance objectives from 
one state with 12,000 performance 
'.'objec Lives from another state. 

What is needed is alevel of 
generality that shows^^^c^fiat is happening 
in developing: 

o course level performance objectives, 

o entry level Job skills, 

tf> and a system of tying them together. 

^This must be in terms of .^a manageable 
list of ten to twenty occupational block 
variahUiS and not in terms of thousands 
specific units. p 

An Example 

Recently, it was possible to compare 
the blocks into which auto body repairs 
(USOE code, 17.0201) was subdivided by 
three states : 



two states which were trying to 
bc'gin the coordination and mutual 
exchange of program objectives 
between var,ious schools 

one state which developed these 
occupational blocks from the 
points of views of both teachers 
and employers. 



'? ive Schools in Two States 

Here are the occupational blocks and 
frequencies in a brief survey of auto 
body=repairs in two states: 



blocks 

painting 

parts assembly 
panel assembly 
frame 



frequency 

3 

2 
2 
2 



r 
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metal repair 
fundamentals of metal 
theory and application 

of welding 
materials for body preparation 

preparation 
hydraulics 
estimating 

glass, \y 
mechanical parts replacement 
spray gun 

used car conditioning 

surfrfices 

chromes 

rust and rot repair 
delivery 
paint types ' 
paint problems 
paint estimating 



A rapid overview of this survey shows 
that it reflects many local issues: 

• some schools work on^ 
production basis where by , students 
are taught to repair^actual cars 
as they come in off the street 

0 not all of these schbols agree on 
the level of complexity indicated . 
by a block (e.g. all the items 
related to painting) o 

• some of the titles seem to stress 
physics (e.g. hydraulics) rather 
than auto body repair 

o attitudes necessary for employment 
are nowhere obvious even though 
onfe could suppose that a trainee 
with a good disposition could 
learn quite a bit through 
on-the-iob training 



t 
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One S t: n t o Corrpnr i ng Toachors nnd Employers 

This study vas done on two levels, the 
block tenninology level and the unit skill 
level. 

1'crmi nolof;y Level 

There vas not too much difference 
between "^teachers and employers when it 
e ;MTje to pinpointing the. names of areas 
nocensary for auco body repair occupational 
training in one state. 

Here is a listing of the nine areas 
teachers nnd employers considered to be the 
main occupational blocks; eight of them were 
oriented tovvords skill and knowledge; the 
ninth was composed of thirteen desirable 
occupational attitudes. 

Welding 

Metal Straightening 
Body and Frame Repair 
Fiber Clacc Body Repair ■ 
Ref inishing 
Glass Replacement 
Panel Replacement 
Estimating 

Thirteen Desirable Occupational Attitudes 

Unit Skill Level 

^nce there was unanimity on the names 
of the necessary occupational blocks, it 
was found helpful to subdivide these 
down into specific units. 

As long as the units rested on the 
name and terminology level, it was still 
possible to find a degree of unanimity. 

Each group was asked to spell out two 
sorts of specifications: 

9 exactly what was to be performed 
^ by the objectives in each- unit 




• exactly how it was to be measured 



The Apparent Differences 

The findings of this survey seemed 
to indicate that : 

e teachers seemed to place greater 
emphasis on the knowledge level, 
of such things as welding than 
employers do 

• employers seemed to require a 
higher level of skill in such 
things as refinishing than did 
teachers. 

0 employers were aware that a student 
with average or above-average [ 
intelligence can learn quite a 
bit through on-the-job training 
if he is willing to learn 

• in many cases, willing to learn 
meint having the right attitude 
as defined by the employer 

One interesting item, "The Right 
Attitude : Learning bn-the-Job," will 
be developed in a separate section. 

Implications of Such Research 

With the assistance provided by an 
item and objective bank, a local educational 
agency will be able to see if it is doing 
a good job meeting the objectives chosen 
by its teachers*. The LEA has then answered 
two questions : 

e "What are we trying to do?" 

• "How well have we done it?" 

Once this first goal of self- 
accountability has been attained, it would 
be possible for the school to ask itself 
some more fundamental questions: 

• "Are the objectives we have 
chosen really relevant to the ■* 
demands of the job market?" 

« "How well do our objectives 

square with the skill ard ability 
patterns required in the market 
place?" T 
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THE 



SYSTEM 



Tying "It AU Together : 

A Few Flowcharts Designed to Sketch out the Big Picture , ^ ^ 
of Modules That Unlvy Educational Targets, Tests» and Technology 



Introductory Comparison to ^-^^.^^ 
CISNE Flowchart 

Training Teachers for Using the ciSNE Concept 
Training Teachers for Using the CISNE Product 
Typical Module (Components 



EVALUATION QUESTIONS 



"VThat are we trying to do?'* 
"How well have we done it?" 
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